Working Paper: Designs of Empirical Evaluations of Non-Experimental Methods in Field Settings

Acknowledgements: This research was supported by a collaborative NSF grant #2015-0285-00. Over the last three decades, a research design has emerged to evaluate the performance of non-experimental designs and design features in field settings. It is called the within-study comparison (WSC) approach, or design replication study. In the traditional WSC design, treatment effects from a randomized experiment are compared to those produced by a non-experimental approach that shares the same target population. The non-experiment may be a quasi-experimental design, such as a regression-discontinuity or an interrupted time series design, or an observational study approach that includes matching methods, standard regression adjustments, and differencein-differences methods. The goals of the WSC are to determine whether the non-experiment can replicate results from a randomized experiment (which provides the causal benchmark estimate), and the contexts and conditions under which these methods work in practice. This paper presents a coherent theory of the design and implementation of WSCs for evaluating non-experimental methods. It introduces and identifies the multiple purposes of WSCs, required design components, common threats to validity, design variants, and causal estimands of interest in WSCs. It highlights two general approaches for empirical evaluations of methods in field settings, WSC designs with independent and dependent benchmark and non-experimental arms. The paper highlights advantages and disadvantages for each approach, and conditions and contexts under which each approach is optimal for addressing methodological questions. EdPolicyWorks